Image processing apparatus, imaging apparatus, image processing method, and program

ABSTRACT

Provided are an apparatus and a method for executing image quality enhancement processing on a fluorescence image. A fluorescence image and a visible light image are input and the image feature amount is extracted, so as to execute pixel value correction processing on the fluorescence image on the basis of a correction parameter determined in accordance with the feature amount. The correction parameter used for pixel value correction is determined by the correction parameter calculation unit on the basis of the feature amount. The image correction unit executes pixel value correction processing that applies the correction parameter determined by the correction parameter calculation unit. For example, blur mode information is obtained as a feature amount from a fluorescence image, and the image correction unit executes pixel value correction processing on the fluorescence image so as to reduce blur of the fluorescence image.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, animaging apparatus, an image processing method, and a program. Thepresent disclosure relates more particularly to an image processingapparatus, an imaging apparatus, an image processing method, and aprogram for performing image processing using a visible image and afluorescence image.

BACKGROUND ART

While visible light images being ordinary color images are used for anendoscope that photographs images of the inside of a living body, theuse of fluorescence images different from visible light images hasadvanced recently.

The fluorescence image is an image, for example, obtained by emittingexcitation light of a specific wavelength region and then photographingfluorescence contained in reflected light from a substance in the livingbody.

The fluorescence images can indicate, for example, an intensitydifference corresponding to each of lesions in the living body. With theuse of fluorescence images, it is possible to effectively performdisease progress status analysis, or the like.

Note that examples of endoscope apparatuses using a visible light imageand a fluorescence image are described in documents such as PatentDocument 1 (Japanese Patent Application Laid-Open No. 2010-82141),Patent Document 2 (Japanese Patent Application Laid-Open No.2011-200330), and Patent Document 3 (Japanese Patent ApplicationLaid-Open No. 2013-248319).

The fluorescence image, however, has a disadvantage of having image blurseverer than in the case of ordinary visible light images. Particularly,an image of a blood vessel or the like located at a deep position in aliving body has a problem that is likely to be unclear due to generationof a great amount of scattered light rays within the living body.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2010-82141-   Patent Document 2: Japanese Patent Application Laid-Open No.    2011-200330-   Patent Document 3: Japanese Patent Application Laid-Open No.    2013-248319

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

The present disclosure is, for example, made in view of the aboveproblems, and aims to provide an image processing apparatus, an imagingapparatus, an image processing method, and a program capable ofobtaining a fluorescence image with little blur, for example.

Solutions to Problems

A first aspect of the present disclosure is an image processingapparatus including:

a feature amount classification processing unit that inputs afluorescence image and a visible light image and extracts a featureamount from at least one of the images; and

an image correction unit that executes pixel value correction processingon the fluorescence image on the basis of a correction parameterdetermined in accordance with the feature amount.

Moreover, a second aspect of the present disclosure is an imagingapparatus including:

an imaging unit that performs imaging processing of a visible lightimage and a fluorescence image, or a visible-fluorescence mixture image;

an image separating unit that inputs a photographed image of the imagingunit, separates a visible light image and a fluorescence image from theinput image and outputs the separated images;

a feature amount classification processing unit that inputs thefluorescence image and the visible light image output by the imageseparating unit and extracts a feature amount from at least one of theimages; and

an image correction unit that executes pixel value correction processingon the fluorescence image output by the image separating unit on thebasis of a correction parameter determined in accordance with thefeature amount.

Moreover, a third aspect of the present disclosure is an imageprocessing method executed in an image processing apparatus, the imageprocessing method including executing:

a feature amount calculation step of executing, by a feature amountclassification processing unit, input of a fluorescence image and avisible light image and extraction of a feature amount from at least oneof the images; and

an image correction step of executing, by an image correction unit,pixel value correction processing on the fluorescence image on the basisof a correction parameter determined in accordance with the featureamount.

Moreover, a fourth aspect of the present disclosure is a program thatcauses an image processing apparatus to execute image processing, theimage processing including processing of:

causing a feature amount classification processing unit to input afluorescence image and a visible light image and extract a featureamount from at least one of the images; and

causing an image correction unit to execute pixel value correctionprocessing on the fluorescence image on the basis of a correctionparameter determined in accordance with the feature amount.

Note that the program of the present disclosure is a program that can beprovided by a storage medium or a communication medium provided in acomputer readable format to an information processing apparatus or acomputer system that can execute various program codes, for example. Byproviding such a program in a computer readable format, processingaccording to the program is implemented on the information processingapparatus or the computer system.

Still other objects, features and advantages of the present disclosurewill be apparent from a detailed description based on exemplaryembodiments of the present disclosure to be described below and attacheddrawings. Note that in the present description, the system represents alogical set of a plurality of apparatuses, and that all the constituentapparatuses need not be in a same housing.

Effects of the Invention

According to a configuration of one exemplary embodiment of the presentdisclosure, it is possible to implement an apparatus and a method toexecute image quality enhancement processing on fluorescence images.

Specifically, the fluorescence image and the visible light image areinput and the image feature amount is extracted, and pixel valuecorrection processing is executed on the fluorescence image on the basisof a correction parameter determined in accordance with the featureamount. The correction parameter used for pixel value correction isdetermined by a correction parameter calculation unit on the basis ofthe feature amount. The image correction unit executes pixel valuecorrection processing that applies the correction parameter determinedby the correction parameter calculation unit. For example, blur modeinformation is obtained as a feature amount from a fluorescence image,and the image correction unit executes pixel value correction processingon the fluorescence image so as to reduce blur of the fluorescenceimage.

This processing enables implementation of an apparatus and a method forexecuting image quality enhancement processing on a fluorescence image.

Note that effects described in the present description are provided forpurposes of exemplary illustration and are not intended to be limiting.Other additional effects may also be contemplated.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating fluorescence images.

FIG. 2 is a diagram illustrating a configuration example of an imageprocessing apparatus.

FIG. 3 is a diagram illustrating configuration and processing of animage processing unit.

FIG. 4 is a diagram illustrating an image feature amount.

FIG. 5 is a diagram illustrating configuration and processing of animage processing unit.

FIG. 6 is a diagram illustrating configuration and processing of animage processing unit.

FIG. 7 is a diagram illustrating configuration and processing of animage processing unit.

FIG. 8 is a diagram illustrating a processing example executed by animage processing unit.

FIG. 9 is a diagram illustrating a processing example executed by animage processing unit.

FIG. 10 is a diagram illustrating configuration and processing of animage processing unit.

FIG. 11 is a diagram illustrating an example of correspondence betweenphotographed images and output images.

FIG. 12 is a diagram illustrating an example of correspondence betweenphotographed images and output images using interpolation images.

FIG. 13 is a diagram illustrating a configuration example of an imageprocessing apparatus.

FIG. 14 is a diagram illustrating configuration and processing of animage processing unit.

FIG. 15 is a diagram illustrating an example of a correspondence with anoutput image generated by image correction based on a photographedimage.

FIG. 16 is a flowchart illustrating a processing sequence executed by animage processing apparatus.

FIG. 17 is a flowchart illustrating a processing sequence executed by animage processing apparatus.

FIG. 18 is a flowchart illustrating a processing sequence executed by animage processing apparatus.

FIG. 19 is a diagram illustrating an example of a hardware configurationof an image processing apparatus.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, details of an image processing apparatus, an imagingapparatus, an image processing method, and a program of the presentdisclosure will be described with reference to the drawings. Note thatthe description will be given in the order of following items.

1. Outline of fluorescence image

2. Configuration and processing of image processing apparatus of thepresent disclosure

3. Configuration for executing image quality enhancement processingapplying blur mode information (PSF information) as image feature amount

4. Configuration for executing image interpolation processing andapplying interpolation image to execute image correction as imagequality enhancement processing

5. Example of correction processing mode of fluorescence image accordingto image photographing sequence

6. Processing sequence executed by image processing apparatus

6-1. Basic sequence of image processing

6-2. Image processing sequence in a configuration executing time sharingphotographing of visible light image and fluorescence image

6-3. Image processing sequence in a configuration of consecutivelyphotographing images according to mode by setting image photographingmodes of visible light images and fluorescence images

7. Hardware configuration example of image processing apparatus

8. Summary of configuration of present disclosure

-   [1. Outline of Fluorescence Image]

First, an outline of a fluorescence image will be described.

As described above, while visible light images being ordinary colorimages are used for an endoscope that photographs images of the insideof a living body, the use of fluorescence images different from visiblelight images has been increasing recently.

A fluorescence image is an image obtained by emitting excitation lightof a specific wavelength and then photographing fluorescence containedin reflected light from a substance in the living body.

The fluorescence images can indicate, for example, an intensitydifference corresponding to each of lesions in the living body. With theuse of fluorescence images, it is possible to effectively performdisease progress status analysis, or the like.

A configuration of photographing a fluorescence image will be describedwith reference to FIG. 1.

A fluorescence image is an image obtained by photographing by emittingexcitation light of a specific wavelength and then inputtingfluorescence output from a living tissue such as blood vessel to animaging element, for example.

FIG. 1 (1) illustrates a configuration example of photographing a bloodvessel 11 located in a relatively shallow portion in a living tissue 10,and (2) illustrates a configuration example of photographing a bloodvessel 11 located in a relatively deep portion in the living tissue 10.

When the excitation light is emitted onto the blood vessel, a pluralityof scattered light rays is generated. In particular, more scatteredlight rays are generated in a deep portion of the living tissue 10, andthis leads to a problem of increasing blur in the fluorescence imagephotographed by the imaging element.

The image processing apparatus according to the present disclosurereduces blur in a fluorescence image for example, making it possible togenerate a fluorescence image with less blur.

Hereinafter, configuration and processing of the image processingapparatus according to the present disclosure will be described indetail.

[2. Configuration and Processing of Image Processing Apparatus ofPresent Disclosure]

Configuration and processing of the image processing apparatus of thepresent disclosure will be described with reference to FIG. 2 and thesubsequent figures.

FIG. 2 is a block diagram illustrating a configuration of an imagingapparatus, as an example of an image processing apparatus 100 accordingto the present disclosure.

Note that the image processing apparatus of the present disclosure isnot limited to the imaging apparatus, and includes an informationprocessing apparatus such as a PC that inputs a photographed image ofthe imaging apparatus and executes image processing, for example.

Hereinafter, configuration and processing of the imaging apparatus willbe described using an example of the image processing apparatus 100according to the present disclosure.

Image processing other than the photographing processing described inthe following exemplary embodiments is not limited to the imagingapparatus, and can be executed in an information processing apparatussuch as a PC.

The image processing apparatus 100 as an imaging apparatus illustratedin FIG. 2 includes a control unit 101, a storage unit 102, a codec 103,an input unit 104, an output unit 105, an imaging unit 106, and an imageprocessing unit 120.

The imaging unit 106 photographs a visible-fluorescence mixture imageincluding both photographing light in a visible light regionconstituting an ordinary color image, and photographing light in afluorescent region.

Alternatively, a visible light image being an ordinary color image and afluorescence image are photographed separately. For example, the twotypes of images are photographed alternately.

As described above, a fluorescence image is an image obtained byphotographing a fluorescent component contained in reflected light froma substance in the living body.

The control unit 101 controls various types of processing to be executedin the imaging apparatus 100, such as image photographing, signalprocessing on a photographed image, image recording processing, anddisplay processing. The control unit 101 includes, for example, a CPUthat executes processing according to various processing programs storedin the storage unit 102 and functions as a data processing unit thatexecutes programs.

The storage unit 102 is constituted with a storage unit of photographedimages, a storage unit for processing program executed by the controlunit 101 and various parameters, a RAM, a ROM, or the like, functioningas working areas at the time of data processing.

The codec 103 executes encoding and decoding processing such ascompression and decompression processing of photographed images.

The input unit 104 is a user operation unit, for example, and inputscontrol information such as start and end of photographing, and variousmode settings.

The output unit 105 is constituted with a display unit, a speaker, orthe like, and is used for display of a photographed image, athrough-the-lens image, or the like, and audio output, or the like.

The image processing unit 120 inputs a photographed image from theimaging unit 106 and executes the image quality enhancement processingon the input image.

Specifically, for example, a corrected visible light image 151 withenhanced image quality and a corrected fluorescence image 152 aregenerated.

Configuration and processing of the image processing unit 120 will bedescribed with reference to FIG. 3 and the subsequent figures.

As illustrated in FIG. 3, the imaging unit 106 according to the presentexemplary embodiment photographs a visible-fluorescence mixture image210 including both photographing light in a visible light regionconstituting an ordinary color image, and photographing light in afluorescent region. The visible-fluorescence mixture image 210photographed by the imaging unit 106 is input to the image processingunit 120.

The image processing unit 120 inputs the visible-fluorescence mixtureimage 210, generates and outputs a corrected visible light image 221 anda corrected fluorescence image 222 that have undergone image qualityenhancement processing.

Processing executed by the image processing unit 120 will be described.

The image processing unit 120 first inputs the visible-fluorescencemixture image 210 photographed by the imaging unit 106 to an imageseparating unit 301, and then, separates the visible-fluorescencemixture image 210 into a visible light image 211 constituted with avisible light component similar to an ordinary RGB color image, and afluorescence image 212 constituted with a fluorescent component alone.

This is executed by matrix operation applying separation matrix, forexample.

The visible light image 211 and the fluorescence image 212 generated byimage separation processing in the image separating unit 301 are inputto the feature amount classification processing unit 302 and the imagecorrection unit 305.

The feature amount classification processing unit 302 inputs the visiblelight image 211 and the fluorescence image 212, extracts an imagefeature amount from these images, executes classification processingbased on the extracted feature amount and stores data into a storageunit (database), while inputting a feature amount data classificationresult to the image correction parameter calculation unit 304.

Note that classification processing is classification processing ingeneral machine learning.

Herein the classification represents classification for determining thecorrection mode and the correction parameter as to what types of imagecorrection is effective for image quality enhancement processing on thebasis of the feature amount obtained from the image.

Note that training data to be applied to this classification is storedin a storage unit (database) 303, and the feature amount classificationprocessing unit 302 uses the training data stored in the storage unit(database) 303 and determines an optimum correction mode or the like forthe image quality enhancement processing for the input image (thevisible light image 211, the fluorescence image 212).

The determination information of the correction mode is input to theimage correction parameter calculation unit 304.

The image correction parameter calculation unit 304 uses the correctionmode determination information input from the feature amountclassification processing unit 302 and training data stored in thestorage unit (database) 303 to determine the image correction parameterto be used for performing image quality enhancement processing on thevisible light image 211 and the fluorescence image 212.

The determined image correction parameter is input to the imagecorrection unit 305.

The image correction unit 305 applies the image correction parameterinput from the image correction parameter calculation unit 304 andexecutes image correction processing on the visible light image 211 andthe fluorescence image 212 and then, generates and outputs the correctedvisible light image 221 and the corrected fluorescence image 222 thathave undergone image quality enhancement processing.

An example of feature amount data obtained by the feature amountclassification processing unit 302 from the visible light image 211 andthe fluorescence image 212 will be described with reference to FIG. 4.

FIG. 4 illustrates an example of the following three types of imagefeature amounts extractable by the feature amount classificationprocessing unit 302 from at least one of the two images.

(1) Point spread function (PSF) (=function indicating a blur mode)

(2) Luminance distribution information

(3) Noise information

“(1) Point spread function (PSF) (=function indicating a blur mode)” isa point spread function (PSF) being a function indicating a blur amountof an image.

As illustrated in the specific example of FIG. 4 (1) (b), this is afunction that indicates the degree of spreading around pixel values at acertain pixel position, that is, a blur amount.

Note that this point spread function is an image feature amountobtainable from either one of the visible light image 211 and thefluorescence image 212.

“(2) Luminance distribution information” is distribution information ofthe luminance value of each pixel in the image. A specific example ofFIG. 4 (2) (b) illustrates a graph (luminance distribution graph)setting pixel positions on the horizontal axis and luminance values onthe vertical axis.

The example illustrated in the figure indicates low luminance value onthe left side of the graph and high luminance values on the right side.Such a luminance distribution is, for example, a luminance distributioncorresponding to an edge region such as a boundary of a subject or thelike.

Note that this type of luminance distribution information is an imagefeature amount obtainable from either one of the visible light image 211and the fluorescence image 212.

“(3) Noise information” is information indicating noise included in animage. An image photographed by the camera contains a certain level ofnoise.

A specific example of FIG. 4 (3) (b) illustrates a graph (noisedistribution graph) setting pixel positions on the horizontal axis andpixel values on the vertical axis.

As illustrated in this graph, the pixel value is a value obtained byadding a predetermined amount of noise to original color or luminance ofthe subject. Note that there are various types of noise such as highfrequency noise, low frequency noise and so on.

Note that this noise information is also an image feature amountobtainable from either one of the visible light image 211 and thefluorescence image 212.

These three types of image feature amounts illustrated in FIG. 4 areexamples of feature amount data obtained from at least one of thevisible light image 211 or the fluorescence image 212 by the featureamount classification processing unit 302.

The feature amount classification processing unit 302 obtains at leastany one of the three types of image feature amounts illustrated in FIG.4 from at least one of the visible light image 211 or the fluorescenceimage 212.

An image correction unit 325 executes image correction processing asimage quality enhancement processing on the visible light image 211 andthe fluorescence image 212 on the basis of the obtained feature amount,and then, generates and outputs the corrected visible light image 221and the corrected fluorescence image 222 with enhanced image quality.

Note that, in the configuration illustrated in FIG. 3, the imaging unit106 is configured to photograph the visible-fluorescence mixture image210 including both the photographing light in the visible light regionconstituting the ordinary color image and the photographing light in thefluorescent region and input the visible-fluorescence mixture image 210to the image processing unit 120.

The image processing unit 120 causes the image separating unit 301 togenerate two images, namely, the visible light image 211 and thefluorescence image 212, from the visible-fluorescence mixture image 210.

Next, with reference to FIG. 5, description will be given onconfiguration and processing of the image processing unit 120 in a casewhere the imaging unit 106 alternately and separately photographs thevisible light image being an ordinary color image, and the fluorescenceimage, and inputs the images to the image processing unit 120.

The imaging unit 106 illustrated in FIG. 5 photographs a visible lightimage 231 corresponding to an ordinary color image and a fluorescenceimage 232 separately and alternately and inputs the images to the imageprocessing unit 120.

The image processing unit 120 sequentially inputs the visible lightimage 231 and the fluorescence image 232, then generates and outputs acorrected visible light image 241 and a corrected fluorescence image 242respectively obtained by applying image quality enhancement processingon each of these images.

Processing executed by the image processing unit 120 will be described.

The visible light image 231 and the fluorescence image 232 sequentiallyphotographed by the imaging unit 106 are input to an image separatingunit 321 of the image processing unit 120.

Under the control of the control unit 101, the image separating unit 321separates an input image from the imaging unit 106 into a visible lightimage 233 and a fluorescence image 234 by time sharing processing.

For example, an image input at a timing t0 is a visible light image, aninput image at a next timing t1 is a fluorescence image, t2=visiblelight image, t3=fluorescence image, and so on.

The control unit 101 controls the image separating unit 321 inaccordance with an image photographing timing of the imaging unit 106and performs processing of separate the visible light image 233 and thefluorescence image 234 from each other.

The visible light image 233 and the fluorescence image 234 generated byimage separation processing on the image separating unit 321 are inputto a feature amount classification processing unit 322 and the imagecorrection unit 325.

The feature amount classification processing unit 322 inputs the visiblelight image 233 and the fluorescence image 234, extracts an imagefeature amount from these images, executes classification processingbased on the extracted feature amount and stores data into a storageunit (database), while inputting a feature amount data classificationresult to an image correction parameter calculation unit 324.

Note that classification processing is classification processing ingeneral machine learning.

Herein, the classification represents classification for determining thecorrection mode as to what types of image correction is effective forimage quality enhancement processing on the basis of the feature amountobtained from the image.

Note that training data to be applied to this classification is storedin a storage unit (database) 323, and the feature amount classificationprocessing unit 322 uses the training data stored in the storage unit(database) 303, and determines an optimum correction mode for the imagequality enhancement processing for the input image (the visible lightimage 233 and the fluorescence image 234).

The determination information of the correction mode is input to theimage correction parameter calculation unit 324.

The image correction parameter calculation unit 324 uses the correctionmode determination information input from the feature amountclassification processing unit 322 and training data stored in thestorage unit (database) 323 to determine the image correction parameterto be used for performing image quality enhancement processing on thevisible light image 233 and the fluorescence image 234.

The determined image correction parameter is input to the imagecorrection unit 325.

The image correction unit 325 applies the image correction parameterinput from the image correction parameter calculation unit 324 toexecute image correction processing on the visible light image 233 andthe fluorescence image 234, and then, generates and outputs thecorrected visible light image 241 and the corrected fluorescence image242 that have undergone image quality enhancement processing.

The feature amount data obtained from the visible light image 233 andthe fluorescence image 234 by the feature amount classificationprocessing unit 322 is the data illustrated in FIG. 4 described aboveand the following data, for example.

(1) Point spread function (PSF) (=function indicating a blur mode)

(2) Luminance distribution information

(3) Noise information

These three image feature amounts illustrated in FIG. 4 are examples offeature amount data obtained from at least one of the visible lightimage 233 or the fluorescence image 234 by the feature amountclassification processing unit 322.

The image correction unit 325 executes image correction processing asimage quality enhancement processing on the visible light image 233 andthe fluorescence image 234 on the basis of the obtained feature amount,and then, generates and outputs the corrected visible light image 241and the corrected fluorescence image 242 with enhanced image quality.

-   [3. Configuration for Executing Image Quality Enhancement Processing    Applying Blur Mode Information (psf Information) as Image Feature    Amount]

As described above with reference to FIG. 4, the feature amountclassification processing unit of the image processing unit illustratedin FIGS. 3 and 5 extracts the following image feature amount from thevisible light image and the fluorescence image, for example.

(1) Point spread function (PSF) (=function indicating a blur mode)

(2) Luminance distribution information

(3) Noise information

The image correction unit of the image processing unit illustrated inFIG. 3 or FIG. 5 executes image correction processing for image qualityenhancement onto a visible light image or a fluorescence image inaccordance with a correction mode or a correction parameter determinedon the basis of these feature amounts.

Hereinafter, a configuration for executing image quality enhancementprocessing applying blur mode information (PSF information) as imagefeature amount will be described.

FIG. 6 is a configuration similar to the configuration of the imageprocessing unit 120 described above with reference to FIG. 3.

That is, the imaging unit 106 photographs the visible-fluorescencemixture image 210 including both the photographing light of the visiblelight region constituting the ordinary color image and the photographinglight of the fluorescent region. The visible-fluorescence mixture image210 photographed by the imaging unit 106 is input to the imageprocessing unit 120.

The image processing unit 120 inputs the visible-fluorescence mixtureimage 210, generates and outputs a corrected visible light image 221 anda corrected fluorescence image 222 that have undergone image qualityenhancement processing.

The image processing unit 120 illustrated in FIG. 6 has a configurationin which the feature amount classification processing unit 302 of theimage processing unit 120 illustrated in FIG. 3 is replaced with a PSFestimation unit 330, and further the image correction parametercalculation unit 304 is replaced with an inverse filter calculation unit340.

Processing executed by the image processing unit 120 illustrated in FIG.6 will be described.

The image processing unit 120 first inputs the visible-fluorescencemixture image 210 photographed by the imaging unit 106 to an imageseparating unit 301, and then, separates the visible-fluorescencemixture image 210 into a visible light image 211 constituted with avisible light component similar to an ordinary RGB color image, and afluorescence image 212 constituted with a fluorescent component alone.

This is executed by matrix operation applying separation matrix, forexample.

The visible light image 211 and the fluorescence image 212 generated byimage separation processing in the image separating unit 301 are inputto the PSF estimation unit 330 and the image correction unit 305.

The PSF estimation unit 330 inputs the visible light image 211 and thefluorescence image 212, extracts a point spread function (PSF) as blurmode information as an image feature amount from these images, andexecutes classification processing based on the extracted blur modeinformation (PSF information), and stores data in the storage unit(database), while inputting a classification result of the blur modeinformation (PSF information) to the inverse filter calculation unit340.

Note that classification processing is classification processing ingeneral machine learning.

Herein, the classification represents classification for determining thecorrection mode as to what types of image correction is effective forimage quality enhancement processing on the basis of the feature amountobtained from the image.

Note that training data to be applied to this classification is storedin the storage unit (database) 303, and the PSF estimation unit 330 usesthe training data stored in the storage unit (database) 303, anddetermines an optimum correction mode for the image quality enhancementprocessing for the input image (the visible light image 211 and thefluorescence image 212).

The determination information of the correction mode is input to theinverse filter calculation unit 340.

The inverse filter calculation unit 340 uses the correction modedetermination information input from the PSF estimation unit 330 and thetraining data stored in the storage unit (database) 303 to generate aninverse filter for performing image quality enhancement processing ofthe visible light image 211 and the fluorescence image 212, that is, aninverse filter such as a Wiener filter, for example, to be applied tosuppress blur.

The generated inverse filter is input to the image correction unit 305.

The image correction unit 305 applies the inverse filter input from theinverse filter calculation unit 340 to execute image correctionprocessing on the visible light image 211 and the fluorescence image212, and then, generates and outputs the corrected visible light image221 and the corrected fluorescence image 222 that have undergone imagequality enhancement processing.

A specific example of processing executed by the PSF estimation unit330, the inverse filter calculation unit 340, and the image correctionunit 305 will be described with reference to FIG. 7.

Note that the example described with reference to FIG. 7 is a processingexample in which the correction target is a fluorescence image.

The PSF estimation unit 330 extracts a point spread function (PSF) (=afunction indicating a mode indicating a blur mode) as the image featureamount from the fluorescence image 212 and outputs the extractedfunction to the inverse filter calculation unit 340.

The inverse filter calculation unit 340 and the image correction unit305 executes processing of generating an inverse filter to be applied toimage correction processing for image quality enhancement for thefluorescence image 212 and pixel value correction processing applyingthe inverse filter on the basis of the point spread function (PSF) (=afunction indicating a mode indicating a blur mode) extracted by the PSFestimation unit 330 from the fluorescence image 212.

FIG. 7 illustrates individual processing to be executed by the PSFestimation unit 330, the inverse filter calculation unit 340, and theimage correction unit 305.

As illustrated in FIG. 7, the PSF estimation unit 330 obtains a pointspread function (PSF) (=a function indicating a mode indicating a blurmode) as an image feature amount from the fluorescence image 212 in stepS11.

The point spread function (PSF) (=function indicating the blur mode) isa function indicating the blur amount of an image as described abovewith reference to FIG. 4 (1).

As illustrated in the specific example of FIG. 4 (1) (b), this is afunction that indicates the degree of spreading around pixel values at acertain pixel position, that is, a blur amount.

Note that herein a point spread function (PSF) is obtained using thefluorescence image 212.

The point spread function (PSF) information extracted from thefluorescence image 212 by the PSF estimation unit 330 is input to theinverse filter calculation unit 340.

The inverse filter calculation unit 340 calculates a coefficientconstituting the inverse filter being a filter for suppressing blur as acorrection parameter to be applied to correction processing on the basisof the point spread function (PSF) information extracted from thefluorescence image 212 by the PSF estimation unit 330 in step S12. Thatis, a multiplication coefficient to be applied to reference pixelssurrounding the correction target pixel is calculated.

An example of correction parameter calculation processing in the inversefilter calculation unit 340 will be described with reference to FIGS. 8and 9.

FIG. 8 includes the following diagrams.

(a1) Example of pixel value distribution on an image before correction

(a2) Example of tap setting and correction parameter (multiplicationcoefficient K_(i))

(b) Example of pixel value distribution on image after correction

(a1) This example of pixel value distribution on an image beforecorrection is an example of pixel value distribution on a fluorescenceimage as a correction target image.

As described above, in a case where an internal image of a living bodyis photographed, for example, the amount of scattered light raysincreases in a fluorescence image, leading to an increase in the bluramount. As illustrated in FIG. 8 (a1), the pixel value distribution is adistribution that gently reflects the luminance of the subject resultingin an image with large amount of blur.

The inverse filter calculation unit 340 performs setting (tap selection)of a reference region for performing image correction of correcting afluorescence image with a large amount of blur like this to be a clearimage with little blur, and further calculates a coefficientconstituting an inverse filter as a filter for blur suppression, thatis, calculates a multiplication coefficient k_(i) to be applied toreference pixels surrounding the correction target pixel.

Specifically, for example, the larger (wider) the blur amount of thefluorescence image 212, the wider range reference pixel region (tapregion) is set so as to determine the multiplication coefficient k_(i)as an effective correction parameter for suppressing the blur of thefluorescence image 212.

“(a2) Example of tap setting and the correction parameter(multiplication coefficient K_(i))” in FIG. 8 illustrates positions ofthe surrounding reference pixels for correcting the pixel value of thecorrection target pixel as the correction target pixel as a center, anda value of multiplication coefficient k_(i) for each of the referencepixels.

The example illustrated in the figure is a case of including 3×3=9pixels arranged as the correction target pixel as a center. 0, −1, and 9illustrated at nine pixel positions are multiplication coefficientsk_(i) being correction parameters calculated by the inverse filtercalculation unit 340. Note that i is a pixel position identifierindicating a pixel position.

In the tap selection processing, the pixel position referred to forcalculating a correction pixel value for the correction target pixel isselected as a tap position. In the example illustrated in the figure, apixel position set to −1 or 9 is a tap.

Moreover, the inverse filter calculation unit 340 calculates themultiplication coefficient k_(i) to be multiplied by the pixel value ofthe tap position. This corresponds to −1 or 9 illustrated in FIG. 8(a2).

Note that the filter calculated by the inverse filter calculation unit340 is a filter for suppressing blur, specifically, for example, aWiener filter or the like is generated.

The inverse filter generated by the inverse filter calculation unit 340is input to the image correction unit 305.

The image correction unit 305 calculates the correction pixel value ofthe fluorescence image 212 using the inverse filter generated by theinverse filter calculation unit 340. Specifically, a correction pixelvalue y of the correction target pixel is calculated by applying thefollowing correction pixel value calculation formula (Formula 1)illustrated in step S13 in FIG. 7.

The correction pixel value y is calculated by the following (Formula 1).

$\begin{matrix}{\left\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 1} \right\rbrack \mspace{335mu}} & \; \\{y = {\sum\limits_{i = 0}^{n}{k_{i} \times x_{i}}}} & \left( {{Formula}\mspace{14mu} 1} \right)\end{matrix}$

Note that In the above (Formula 1), each of symbols has the followingmeaning.

y: correction pixel value of correction target pixel

x_(i): pixel value of reference pixel

i: pixel identifier of reference pixel

k_(i): multiplication coefficient corresponding to reference pixel i

The correction target pixel is a pixel at a center position of 3×3=9pixels illustrated in FIG. 8 (a2), for example.

The reference pixel is each of pixels of 3×3=9 pixels, and x_(i) is apixel value of each of the pixels. i is the identifier of the pixel. Inthe case nine pixels are referred to, n=8 is set, and the correctionpixel value T is calculated using the pixel value of each of the pixelsof i=0 to 8.

k_(i) is a multiplication coefficient for the pixel value x_(i) set ateach of pixel positions i.

Pixel values of the correction target pixels are calculated inaccordance with the above (Formula 1).

Note that the tap setting and the correction parameter (multiplicationcoefficient) setting illustrated in FIG. 8 (a2) are mere examples, andthe tap and the correction parameter are changed to various settings inaccordance with the feature amount.

The image correction unit 305 sequentially calculates the correctionpixel values of all of the constituent pixels of the fluorescence image212 in accordance with the above-described (Formula 1), and generatesand outputs the calculated configured corrected fluorescence image 222.

An example of pixel value distribution of the corrected fluorescenceimage 222 is illustrated in FIG. 8 (b).

The pixel value distribution of the corrected fluorescence image 222 isan image with steeper gradient of the pixel value change with suppressedblur, as compared with the pixel value distribution of the fluorescenceimage before correction illustrated in FIG. 8 (a1).

This is a result of performing pixel value correction using an inversefilter with a coefficient set as a blur suppression filter.

In this manner, the pixel value of the fluorescence image is correctedusing the PSF information being the feature amount indicating the blurmode of the fluorescence image, making it possible to perform imagequality enhancement on a fluorescence image having a large amount ofblur, that is, possible to generate and output the correctedfluorescence image 222 with reduced blur amount.

Note that the examples illustrated in FIGS. 8 (a1) and (a2) are examplesof tap setting and a setting of correction parameter (multiplicationcoefficient k_(i)) in a case where the expansion of blur is relativelysmall.

The tap setting and the setting of the correction parameter(multiplication coefficient k_(i)) are changed in accordance with thePSF obtained as the feature amount, that is, the blur mode.

FIG. 9 illustrates examples of tap setting and the setting of thecorrection parameter (multiplication coefficient K_(i)) in a case wherethe expansion of blur is relatively large.

As illustrated in FIGS. 9 (a1) and (a2), in a case where the blur amountis large, the tap setting is performed to increase the pixel region tobe used as reference determination, so as to perform processing ofdetermining the correction pixel value on the basis of the pixel valueof the reference pixel in the wider range.

In this manner, the reference pixel region selection processing appliedto the correction processing, that is, the tap selection processing isexecuted, and correction parameters (multiplication coefficients) arecalculated to perform correction processing on the basis of the pointspread function (PSF) information calculated by the PSF estimation unit330 set as the feature amount classification processing unit 302, makingit possible to perform optimum pixel value correction according to theblur mode and to generate a high quality corrected fluorescence imagewith reduced blur.

Next, referring to FIG. 10, description will be given on configurationand processing of the image processing unit 120 in a case where theimaging unit 106 photographs the visible light image being an ordinarycolor image and a fluorescence image separately and alternately andinputs the images to the image processing unit 120.

The configuration of the image processing apparatus illustrated in FIG.10 is similar to the configuration of the image processing apparatuswith reference to FIG. 5.

The imaging unit 106 illustrated in FIG. 10 photographs the visiblelight image 231 corresponding to ordinary color image and thefluorescence image 232 separately and alternately and inputs the imagesto the image processing unit 120.

The image processing unit 120 sequentially inputs the visible lightimage 231 and the fluorescence image 232, then generates and outputs acorrected visible light image 241 and a corrected fluorescence image 242respectively obtained by applying image quality enhancement processingon each of these images.

The image processing unit 120 illustrated in FIG. 10 has a configurationin which the feature amount classification processing unit 322 of theimage processing unit 120 illustrated in FIG. 5 is replaced with a PSFestimation unit 350, and furthermore the image correction parametercalculation unit 324 is replaced with an inverse filter calculation unit360.

Processing executed by the image processing unit 120 illustrated in FIG.10 will be described.

The visible light image 231 and the fluorescence image 232 sequentiallyphotographed by the imaging unit 106 are input to an image separatingunit 321 of the image processing unit 120.

Under the control of the control unit 101, the image separating unit 321separates an input image from the imaging unit 106 into a visible lightimage 233 and a fluorescence image 234 by time sharing processing.

An example of setting is: an image input at a timing t0 is a visiblelight image, an input image at a next timing t1 is a fluorescence image,t2 =visible light image, t3=fluorescence image, and so on.

The control unit 101 controls the image separating unit 321 inaccordance with an image photographing timing of the imaging unit 106and performs processing of separate the visible light image 233 and thefluorescence image 234 from each other.

The visible light image 233 and the fluorescence image 234 generated byimage separation processing in the image separating unit 301 are inputto the PSF estimation unit 350 and the image correction unit 325.

The PSF estimation unit 350 inputs the visible light image 233 and thefluorescence image 234, extracts a point spread function (PSF) as blurmode information as an image feature amount from these images, andexecutes classification processing based on the extracted blur modeinformation (PSF information), and stores data in the storage unit(database), while inputting a classification result of the blur modeinformation (PSF information) to the inverse filter calculation unit360.

Note that classification processing is classification processing ingeneral machine learning.

Herein, the classification represents classification for determining thecorrection mode as to what types of image correction is effective forimage quality enhancement processing on the basis of the feature amountobtained from the image.

Note that training data to be applied to this classification is storedin the storage unit (database) 323, and the PSF estimation unit 350 usesthe training data stored in the storage unit (database) 323, anddetermines an optimum correction mode for the image quality enhancementprocessing for the input image (the visible light image 233 and thefluorescence image 234).

The determination information of the correction mode is input to theinverse filter calculation unit 360.

The inverse filter calculation unit 360 uses the correction modedetermination information input from the PSF estimation unit 350 and thetraining data stored in the storage unit (database) 323 to generate aninverse filter for performing image quality enhancement processing ofthe visible light image 233 and the fluorescence image 234, that is, aninverse filter such as a Wiener filter, for example, to be applied tosuppress blur.

The generated inverse filter is input to the image correction unit 325.

The image correction unit 325 applies the inverse filter input from theinverse filter calculation unit 360 to execute image correctionprocessing on the visible light image 233 and the fluorescence image234, and then, generates and outputs the corrected visible light image241 and the corrected fluorescence image 242 that have undergone imagequality enhancement processing.

A specific example of processing executed by the PSF estimation unit350, the inverse filter calculation unit 360, and the image correctionunit 325 will be described with reference to the processing steps S21 toS23 illustrated in FIG. 10.

Note that the example described with reference to FIG. 10 is aprocessing example in which the correction target is a fluorescenceimage.

FIG. 10 illustrates processing to be executed by the PSF estimation unit350, the inverse filter calculation unit 360, and the image correctionunit 325, as steps S21 to S23, respectively.

As illustrated in FIG. 10, the PSF estimation unit 350 obtains a pointspread function (PSF) (=a function indicating a mode indicating a blurmode) as an image feature amount from the fluorescence image 234 in stepS21.

The point spread function (PSF) (=function indicating the blur mode) isa function indicating the blur amount of an image as described abovewith reference to FIG. 4 (1).

As illustrated in the specific example of FIG. 4 (1) (b), this is afunction that indicates the degree of spreading around pixel values at acertain pixel position, that is, a blur amount.

Note that herein a point spread function (PSF) is obtained using thefluorescence image 234.

The point spread function (PSF) information extracted from thefluorescence image 234 by the PSF estimation unit 350 is input to theinverse filter calculation unit 360.

The inverse filter calculation unit 360 calculates a coefficientconstituting the inverse filter being a filter for suppressing blur as acorrection parameter to be applied to correction processing on the basisof the point spread function (PSF) information extracted from thefluorescence image 234 by the PSF estimation unit 350 in step S22. Thatis, a multiplication coefficient to be applied to reference pixelssurrounding the correction target pixel is calculated.

The correction parameter calculation processing in the inverse filtercalculation unit 360 is similar to that described above with referenceto FIGS. 8 and 9.

Note that the filter calculated by the inverse filter calculation unit360 is a filter for suppressing blur, specifically, for example, aWiener filter or the like is generated.

The inverse filter generated by the inverse filter calculation unit 360is input to the image correction unit 325.

The image correction unit 325 calculates the correction pixel value ofthe fluorescence image 234 using the inverse filter generated by theinverse filter calculation unit 360. Specifically, the correction pixelvalue y of the correction target pixel is calculated by applying thefollowing correction pixel value calculation formula (Formula 2)illustrated in Step S23 of FIG. 10.

The correction pixel value y is calculated by the following (Formula 2).

$\begin{matrix}{\left\lbrack {{Mathematical}\mspace{14mu} {Expression}\mspace{14mu} 2} \right\rbrack \mspace{340mu}} & \; \\{y = {\sum\limits_{i = 0}^{n}{k_{i} \times x_{i}}}} & \left( {{Formula}\mspace{14mu} 2} \right)\end{matrix}$

Note that in the above (Formula 2), each of symbols has the followingmeaning.

y: correction pixel value of correction target pixel

x_(i): pixel value of reference pixel

i: pixel identifier of reference pixel

k_(i): multiplication coefficient corresponding to reference pixel i

The correction target pixel is a pixel at a center position of 3×3=9pixels illustrated in FIG. 8 (a2), for example.

The reference pixel is each of pixels of 3×3=9 pixels, and x_(i) is apixel value of each of the pixels. i is the identifier of the pixel. Inthe case nine pixels are referred to, n=8 is set, and the correctionpixel value T is calculated using the pixel value of each of the pixelsof i=0 to 8.

k_(i) is a multiplication coefficient for the pixel value x_(i) set ateach of pixel positions i.

Pixel values of the correction target pixels are calculated inaccordance with the above (Formula 2).

In this manner, the reference pixel region selection processing appliedto the correction processing, that is, the tap selection processing isexecuted, and correction parameters (multiplication coefficients) arecalculated to perform correction processing on the basis of the pointspread function (PSF) information calculated by the PSF estimation unit350 set as the feature amount classification processing unit 302, makingit possible to perform optimum pixel value correction according to theblur mode and to generate a high quality corrected fluorescence imagewith reduced blur.

Note that the processing example described with reference to FIGS. 6 to10 focused on the blur suppression processing on the fluorescence image,while it is also possible to suppress the blur of the visible lightimage with application of similar processing stages to the visible lightimage.

-   [4. Configuration for Executing Image Interpolation Processing and    Applying Interpolation Image to Execute Image Correction as Image    Quality Enhancement Processing]

Next, a configuration for executing image interpolation processing andapplying interpolation image to execute image correction as imagequality enhancement processing will be described.

The imaging unit 106 of the image processing apparatus 100 illustratedin FIG. 2 photographs different types of images such as a visible lightimage, a fluorescence image, or a visible light+fluorescence image.

In a configuration in which the imaging unit 106 executes photographingof different types of images, frame rates enabling outputting a visiblelight image and a fluorescence image do not always match.

For example, here is assumed a configuration in which the imaging unit106 alternately photographs the following two types of different images.

(a) visible light image

(b) visible light+fluorescence image Specifically, the configuration hasphotographed image settings illustrated in FIG. 11 (1).

The time-series sequence of the photographed image illustrated in FIG.11 (1) is as follows.

At time t1, an image (f1) of visible light+fluorescence is photographed.

At time t2, a visible light image (f2) is photographed.

At time t3, an image (f3) of visible light+fluorescence is photographed.

At time t4, a visible light image (f4) is photographed.

At time t5, an image (f5) of visible light+fluorescence is photographed.

At time t6, a visible light image (f6) is photographed. This sequence isrepeated thereafter.

When photographing of images is performed with this setting, the imagethat can be output would be as in the setting of the output images inFIG. 11 (2). Specifically, the following settings are applied.

At time t1, the visible light image (f1) and a fluorescence image (f1)is obtained by image separation processing of the image (f1) of visiblelight+fluorescence.

At time t2, a visible light image (f2) alone is output with no output ofa fluorescence image.

At time t3, a visible light image (f3) and a fluorescence image (f3) isobtained by image separation processing of the image (f3) of visiblelight+fluorescence.

At time t4, a visible light image (f4) alone is output with no output ofa fluorescence image.

At time t5, a visible light image (f5) and a fluorescence image (f5) isobtained by image separation processing of the image (f5) of visiblelight+fluorescence.

At time t6, a visible light image (f6) alone is output with no output ofa fluorescence image.

This sequence is repeated thereafter.

With image photographing processing illustrated in FIG. 11, the framerate of the output fluorescence image would be half the frame rate ofthe visible light image.

With reference to FIG. 12 and the subsequent figures, configuration andprocessing of an image processing apparatus that prevents such areduction in frame rate and further executes image quality enhancementprocessing will be described.

FIG. 12 is a diagram illustrating image interpolation processingexecuted by the image processing apparatus of the present exemplaryembodiment.

FIG. 12 (1) illustrates the setting of the photographed image similar tothat in FIG. 11 (1), and the time-series sequence of the photographedimage is as follows.

At time t1, an image (f1) of visible light+fluorescence is photographed.

At time t2, a visible light image (f2) is photographed.

At time t3, an image (f3) of visible light+fluorescence is photographed.

At time t4, a visible light image (f4) is photographed.

At time t5, an image (f5) of visible light+fluorescence is photographed.

At time t6, a visible light image (f6) is photographed.

This sequence is repeated thereafter.

In the configuration of the present exemplary embodiment, as illustratedin the output image of FIG. 12 (2), interpolation fluorescence imagesare generated at each of times, at timings t2, t4, t6 . . . at which nofluorescence image is photographed.

The images correspond to an interpolation fluorescence image (t2), aninterpolation fluorescence image (t4), and an interpolation fluorescenceimage (t6) illustrated in FIG. 12 (2).

With this image interpolation processing, it is possible to obtain anoutput image illustrated in FIG. 12 (2). Specifically, the followingsettings are applied.

At time t1, the visible light image (f1) and a fluorescence image (f1)is obtained by image separation processing of the image (f1) of visiblelight+fluorescence.

At time t2, a visible light image (f2) and an interpolation fluorescenceimage (f2) are output.

At time t3, a visible light image (f3) and a fluorescence image (f3) isobtained by image separation processing of the image (f3) of visiblelight+fluorescence.

At time t4, a visible light image (f4) and an interpolation fluorescenceimage (f4) are output.

At time t5, a visible light image (f5) and a fluorescence image (f5) isobtained by image separation processing of the image (f5) of visiblelight+fluorescence.

At time t6, a visible light image (f6) and an interpolation fluorescenceimage (f6) are output.

This sequence is repeated thereafter.

With execution of image interpolation processing in this manner, it ispossible to set the frame rate of the output fluorescence image to bethe same as the frame rate of the visible light image.

A configuration example and processing of an image processing apparatushaving an image processing unit that performs image interpolationprocessing and image quality enhancement processing will be describedwith reference to FIG. 13.

The imaging unit 106 of an image processing apparatus 400 illustrated inFIG. 13 photographs an image with the photographing sequence of thephotographed image described with reference to FIG. 12 (1).

FIG. 13 illustrates the following three time-series photographed imageframes.

(a) Photographed image frame at time t (n−1)=an image (Fn−1) 421 ofvisible light+fluorescence,

(b) Photographed image frame at time t (n)=visible light image (Fn) 422,and

(c) Photographed image frame at time t (n+1)=an image (Fn+1) 423 ofvisible light+fluorescence.

These three consecutive image frames 421 to 423 are input to aninterpolation image generation unit 410.

The interpolation image generation unit 410 includes a motion estimationprocessing (ME) unit 411, an interpolation image estimation unit 412,and a subtraction unit 413.

The motion estimation processing (ME) unit 411 inputs three image frames421 to 423, performs motion estimation (ME) between these image frames,and inputs motion information to the interpolation image estimation unit412.

The three image frames 421 to 423 are images photographed in accordancewith a photographing sequence of a photographed pixel eyebrow of FIG.12(1), and are photographed at different photographing timings.Accordingly, a shift occurs in the position of the subject includedbetween individual images due to motion, blur or the like of thesubject.

The motion estimation processing (ME) unit 411 calculates the amount ofsubject position shift between the three image frames 421 to 423 andinputs this as motion information to the interpolation image estimationunit 412.

The interpolation image estimation unit 412 inputs three image frames421 to 423 and inputs motion information of these three images from themotion estimation processing (ME) unit 411.

The interpolation image estimation unit 412 applies the motioninformation of the three images input from the motion estimationprocessing (ME) unit 411, and performs alignment of the three imageframes 421 to 423. That is, this corresponds to execution of alignmentprocessing of suppressing the shift so as to locate a same subject on asame coordinate position of each of the images.

Note that in this alignment processing, the photographed image frame attime t(n)=visible light image (Fn) 422 is set as a reference frame, andprocessing of achieving a match between subject position of the visiblelight image photographed at this time t(n) with the subject position ofanother image frame is executed.

The interpolation image estimation unit 412 blends the three imageframes 421 to 423 after the alignment and generates an image (Fn) ofvisible light+fluorescence, which is a virtual photographed image frameat time t(n).

The image (Fn) of visible light+fluorescence generated by theinterpolation image estimation unit 412 is input to the subtraction unit413.

The subtraction unit 413 perform processing of subtracting the visiblelight image (Fn) 422 from the image (Fn) of visible light+fluorescencegenerated by the interpolation image estimation unit 412.

This subtraction processing is used to generate an interpolationfluorescence image (Fn) 431.

The interpolation fluorescence image (Fn) 431 is a virtual imagecorresponding to the fluorescence image photographed at thephotographing timing of time t (n) in which imaging of the fluorescenceimage is not actually executed.

An interpolation fluorescence image (Fn) 431 which is an estimated imageat the photographing timing of the time t(n) and a visible light image(Fn) 422 which is actually photographed at the imaging timing of thetime t(n) are input to an image processing unit 450.

Configuration and processing of the image processing unit 450 will bedescribed with reference to FIG. 14.

The configuration of the image processing unit 450 illustrated in FIG.14 corresponds to a configuration in which the image separating unit isomitted from the image processing unit 120 described with reference toFIG. 3, FIG. 5, or the like.

Processing executed by the image processing unit 450 will be described.

The visible light image 422 input from the imaging unit 106 and theinterpolation fluorescence image 431 generated by the interpolationimage generation unit 410 are input to a feature amount classificationprocessing unit 451 and an image correction unit 454 of the imageprocessing unit 450.

The feature amount classification processing unit 451 inputs the visiblelight image 422 and the interpolation fluorescence image 431, extractsan image feature amount from these images, executes classificationprocessing based on the extracted feature amount and stores data into astorage unit (database), while inputting a feature amount dataclassification result to an image correction parameter calculation unit453.

Note that classification processing is classification processing ingeneral machine learning.

Herein, the classification represents classification for determining thecorrection mode as to what types of image correction is effective forimage quality enhancement processing on the basis of the feature amountobtained from the image.

Note that training data to be applied to this classification is storedin the storage unit (database) 452, and the feature amountclassification processing unit 451 uses the training data stored in thestorage unit (database) 303, and determines an optimum correction modefor the image quality enhancement processing for the input image (thevisible light image 422, the interpolation fluorescence image 431).

The determination information of the correction mode is input to theimage correction parameter calculation unit 453.

The image correction parameter calculation unit 453 uses the correctionmode determination information input from the feature amountclassification processing unit 451 and training data stored in thestorage unit (database) 452 to determine the image correction parameterto be used for performing image quality enhancement processing on thevisible light image 422 and the interpolation fluorescence image 431.

The determined image correction parameter is input to the imagecorrection unit 454.

The image correction unit 454 applies the image correction parameterinput from the image correction parameter calculation unit 453 toexecute image correction processing on the visible light image 422 andthe interpolation fluorescence image 431, and then, generates andoutputs a corrected visible light image 471 and a corrected fluorescenceimage 472, to which the image quality enhancement processing has beenapplied.

The feature amount data obtained from the visible light image 422 andthe interpolation fluorescence image 431 by the feature amountclassification processing unit 451 is the data illustrated in FIG. 4described above and the following data, for example.

(1) Point spread function (PSF) (=function indicating a blur mode)

(2) Luminance distribution information

(3) Noise information

These three image feature amounts illustrated in FIG. 4 are examples offeature amount data obtained from at least one of the visible lightimage 422 or the interpolation fluorescence image 431 by the featureamount classification processing unit 451.

The image correction unit 454 executes image correction processing asimage quality enhancement processing on the visible light image 422 andthe interpolation fluorescence image 431 on the basis of the obtainedfeature amount so as to generate and outputs the corrected visible lightimage 471 and the corrected fluorescence image 472 with enhanced imagequalities.

-   [5. Example of Correction Processing Mode of Fluorescence Image    According to Image Photographing Sequence]

Next, with reference to FIG. 15, an example of a correction processingmode of a fluorescence image according to an image photographingsequence will be described.

As described above with reference to FIGS. 11 and 12, for example, theimaging unit 106 of the image processing apparatus 100 illustrated inFIG. 2 is configured to photograph different types of images, such as avisible light image, a fluorescence image, or a visiblelight+fluorescence image. These image photographing timing and sequencesare assumed to have various settings.

As an example, a photographing sequence as illustrated in a photographedimage in FIG. 15 (1) is also assumed.

The sequence of the photographed image illustrated in FIG. 15 (1) is animage photographing sequence of alternately photographing a plurality ofvisible light image frames and a plurality of fluorescence image frames.

In such an image photographing sequence, for example, the image qualityenhancement processing on the fluorescence image (f4) can be performedwith application of the configuration of the image processing unitdescribed above with reference to FIG. 5, and with application of thevisible light image (f3) being an immediately preceding photographedimage frame.

Each of the three image frames of the fluorescence image (f5) to thefluorescence image (f7), however, has no visible light imagephotographed immediately before the photographing timing of these threeimage frames.

Accordingly, this makes it difficult to perform image qualityenhancement processing applying the two consecutively photographed imageframes described with reference to FIG. 5.

In such a case, image correction processing as illustrated in FIG. 15 isexecuted.

Specifically, the image correction mode determined by an image featureamount classification processing unit 501 on the basis of the visiblelight image (f3) and the fluorescence image (f4) and the correctionparameter determined by an image correction parameter calculation unit502 are applied so as to execute image correction not merely on theimage (f4) but also on the fluorescence image (f5) to the fluorescenceimage (f7) consecutively photographed thereafter.

With this processing, as illustrated in FIG. 15 (2), it is possible togenerate and output the corrected fluorescence image (f4) to thecorrected fluorescence image (f7) as output images.

That is, it is possible to generate and output a corrected fluorescenceimage with higher image quality even when the visible light image andthe fluorescence image are not alternately photographed.

-   [6. Processing Sequence Executed by Image Processing Apparatus]

Next, a processing sequence executed by the image processing apparatusaccording to the present disclosure will be described with reference toa flowchart illustrated in FIG. 16 and subsequent figures.

-   [6-1. Basic Sequence of Image Processing]

First, a basic sequence executed by the image processing apparatusaccording to the present disclosure will be described with reference toa flowchart illustrated in FIG. 16.

FIG. 16 is a flowchart illustrating a basic processing sequence executedin the image processing apparatus.

The processing according to the flow illustrated in FIG. 16 is executedunder the control of a control unit having a program execution functionin accordance with a program stored in the storage unit of the imageprocessing apparatus, for example.

Hereinafter, processing of each of steps of the flow illustrated in FIG.16 will be sequentially described.

-   (Steps S101 to S102)

Steps S101 to S102 correspond to image photographing processing.

This is, for example, image photographing processing in the imaging unit106 illustrated in FIGS. 2, 3, 5, or the like.

The photographed image is executed by photographing processing of avisible light image and a fluorescence image, or a visible-fluorescencemixture image.

Emission from light source in step S101 includes visible light emissionprocessing for photographing a visible light image and excitation lightemission processing for photographing a fluorescence image.

The photographing processing in step S102 is image photographingprocessing under this emission from light source, and is photographingprocessing of a visible light image and a fluorescence image, or avisible-fluorescence mixture image.

-   (Step S103)

Processing in steps S103 to S106 is image processing in the imageprocessing unit illustrated in FIGS. 2, 3, 5, or the like, for example.

Step S103 executes processing of separating the visible light image andthe fluorescence image from each other.

This processing is executed in for example, the image separating unit301 in FIG. 3 and the image separating unit 321 illustrated in FIG. 5.

In the example illustrated in FIG. 3, the visible-fluorescence mixtureimage 210 photographed by the imaging unit 106 is input to an imageseparating unit 301, and then, the visible-fluorescence mixture image210 is separated into a visible light image 211 constituted with avisible light component similar to an ordinary RGB color image, and afluorescence image 212 constituted with a fluorescent component alone.

This is executed by matrix operation applying separation matrix, forexample.

In the example illustrated in FIG. 5, the visible light image 231 andthe fluorescence image 232 sequentially photographed by the imaging unit106 are input to the image separating unit 321 of the image processingunit 120.

Under the control of the control unit 101, the image separating unit 321separates an input image from the imaging unit 106 into a visible lightimage 233 and a fluorescence image 234 by time sharing processing.

-   (Step S104)

Next, feature amount extraction from the image is performed in stepS104.

This processing is processing to be executed by the feature amountclassification processing unit illustrated in FIGS. 3 and 5, forexample.

The feature amount classification processing unit inputs the visiblelight image and the fluorescence image, extracts an image feature amountfrom these images, executes classification processing based on theextracted feature amount and stores data into a storage unit (database),while inputting a feature amount data classification result to the imagecorrection parameter calculation unit.

Note that the feature amount extracted from the image by the featureamount classification processing unit is, for example, a feature amountsuch as a point spread function (PSF) (=a function indicating a blurmode) being blur mode information.

Note that other feature amounts include, for example, luminancedistribution information and noise information, or the like as describedabove with reference to FIG. 4.

-   (Step S105)

Next, step S105 executes correction parameter calculation processing.

This processing is processing executed by the image correction parametercalculation unit illustrated in FIGS. 3 and 5, for example.

The image correction parameter calculation unit uses the correction modedetermination information input from the feature amount classificationprocessing unit and training data stored in the storage unit (database)to determine the image correction parameter to be used for performingimage quality enhancement processing on the visible light image and thefluorescence image.

The determined image correction parameter is input to the imagecorrection unit.

Note that one specific example of the correction parameter calculated bythe image correction parameter calculation unit is a multiplicationcoefficient being a setting parameter of an inverse filter forsuppressing blur.

The image correction parameter calculation unit generates an inversefilter such as the Wiener filter to be applied for suppressing blur.

-   (Step S106)

Finally, step S106 executes image correction processing. This processingis processing executed by the image correction unit illustrated in FIGS.3 and 5, for example.

The image correction unit applies the image correction parameter inputfrom the image correction parameter calculation unit to execute imagecorrection processing on the visible light image and the fluorescenceimage, and then, generates and outputs the corrected visible light imageand the corrected fluorescence image that have undergone image qualityenhancement processing.

Note that a specific example of the correction processing is pixel valuecorrection processing that utilizes an inverse filter such as a Wienerfilter that executes blur suppression processing.

For example, the blur suppression processing described above withreference to FIGS. 6 to 10 is executed.

The corrected image having blur suppressed and image quality enhanced bythis image correction processing is output.

Note that the processing example described with reference to FIGS. 6 to10 focused on the blur suppression processing on the fluorescence image,while it is also possible to suppress the blur of the visible lightimage with application of similar processing stages to the visible lightimage.

-   [6-2. Image Processing Sequence in a Configuration Executing Time    Sharing Photographing of Visible Light Image and Fluorescence Image]

Next, an image processing sequence in a configuration executing timesharing photographing of a visible light image and a fluorescence imagewill be described with reference to a flowchart illustrated in FIG. 17.

This processing sequence is processing sequence, for example, asillustrated in a configuration in FIGS. 5 and 10 described above, inwhich the imaging unit 106 alternately photographs a visible light imageand a fluorescence image and inputs the images to the image processingunit to execute image processing.

The processing according to the flow illustrated in FIG. 17 is executedunder the control of a control unit having a program execution functionin accordance with a program stored in the storage unit of the imageprocessing apparatus, for example.

Hereinafter, processing of each of steps of the flow illustrated in FIG.17 will be sequentially described.

-   (Step S201)

First, the control unit judges in step S201 whether it is a timing ofphotographing a visible light image or a timing of photographing afluorescence image.

In a case where determination is the photographing timing of the visiblelight image, the processing proceeds to step S202.

In contrast, in a case where determination is the photographing timingof the fluorescence image, the processing proceeds to step S204.

-   (Steps S202 to S203)

In a case where the control unit determined in step S201 that it is thetiming of photographing the visible light image, the imaging unitperforms in steps S202 to S203 light emission processing necessary forvisible light image photographing and visible light image photographingprocessing under the control of the control unit.

-   (Steps S204 to S205)

In contrast, in a case where the control unit determined in step S201that it is the timing of photographing the fluorescence image, theimaging unit performs in steps S204 to S205 light emission processingnecessary for fluorescence image photographing and fluorescence imagephotographing processing under the control of the control unit.

-   (Step S206)

Next, the control unit determines in step S206 whether or notphotographing of an image pair of a visible light image and afluorescence image has been completed.

In a case where photographing of the image pair of the visible lightimage and the fluorescence image has not been completed, the processingreturns to step S201, and the processing of step S201 and the subsequentsteps is repeated.

In contrast, in a case where photographing of the image pair of thevisible light image and the fluorescence image has been completed, theprocessing proceeds to step S207.

-   (Step S207)

Processing in steps S207 to S209 is image processing to be executed inthe image processing unit illustrated in FIGS. 5, 10, or the like, forexample.

In step S207, feature amount extraction from the image is performed.

This processing is processing to be executed by the feature amountclassification processing unit illustrated in FIG. 5, for example.

The feature amount classification processing unit inputs the visiblelight image and the fluorescence image, extracts an image feature amountfrom these images, executes classification processing based on theextracted feature amount and stores data into a storage unit (database),while inputting a feature amount data classification result to the imagecorrection parameter calculation unit.

Note that the feature amount extracted from the image by the featureamount classification processing unit is, for example, a feature amountsuch as a point spread function (PSF) (=a function indicating a blurmode) being blur mode information.

Note that other feature amounts include, for example, luminancedistribution information and noise information, or the like as describedabove with reference to FIG. 4.

-   (Step S208)

Next, step S208 executes correction parameter calculation processing.

This processing is processing executed by the image correction parametercalculation unit illustrated in FIG. 5, for example.

The image correction parameter calculation unit uses the correction modedetermination information input from the feature amount classificationprocessing unit and training data stored in the storage unit (database)to determine the image correction parameter to be used for performingimage quality enhancement processing on the visible light image and thefluorescence image.

The determined image correction parameter is input to the imagecorrection unit.

Note that one specific example of the correction parameter calculated bythe image correction parameter calculation unit is a multiplicationcoefficient being a setting parameter of an inverse filter forsuppressing blur.

The image correction parameter calculation unit generates an inversefilter such as the Wiener filter to be applied for suppressing blur.

-   (Step S209)

Finally, step S209 executes image correction processing.

This processing is, for example, processing executed by the imagecorrection unit illustrated in FIG. 5.

The image correction unit applies the image correction parameter inputfrom the image correction parameter calculation unit to execute imagecorrection processing on the visible light image and the fluorescenceimage, and then, generates and outputs the corrected visible light imageand the corrected fluorescence image that have undergone image qualityenhancement processing.

Note that a specific example of the correction processing is pixel valuecorrection processing that utilizes an inverse filter such as a Wienerfilter that executes blur suppression processing.

For example, the blur suppression processing described above withreference to FIGS. 6 to 10 is executed.

The corrected image having blur suppressed and image quality enhanced bythis image correction processing is output.

Note that the processing example described with reference to FIGS. 6 to10 focused on the blur suppression processing on the fluorescence image,while it is also possible to suppress the blur of the visible lightimage with application of similar processing stages to the visible lightimage.

-   [6-3. Image Processing Sequence in a Configuration of Consecutively    Photographing Images According to Mode by Setting Image    Photographing Modes of Visible Light Images and Fluorescence Images]

Next, an image processing sequence in a configuration of performingconsecutive photographing of an image according to modes by setting eachof image photographing modes of the visible light image and thefluorescence image will be described with reference to a flowchartillustrated in FIG. 18.

This processing sequence is, for example, a processing sequence in thecase of executing the image photographing processing described abovewith reference to FIG. 15.

In other words, the flow illustrated in FIG. 18 is a flow illustratingprocessing in a case where the setting enables a visible light imagephotographing mode that executes consecutive photographing of a visiblelight image and a fluorescence image photographing mode that executesconsecutive photographing of a fluorescence image to be alternatelyswitched, as illustrated in the image sequence of photographed images inFIG. 15 (1).

The processing according to the flow illustrated in FIG. 18 is executedunder the control of the control unit having a program executionfunction in accordance with a program stored in the storage unit of theimage processing apparatus, for example.

Hereinafter, processing of each of steps of the flow illustrated in FIG.18 will be sequentially described.

-   (Step S301)

First, in step S301, the control unit judges whether the currentphotographing mode is the visible light image photographing mode thatexecutes consecutive photographing of a visible light image or thefluorescence image photographing mode that executes consecutivephotographing of a fluorescence image.

In a case where determination is the visible light image photographingmode, the processing proceeds to step S302.

In contrast, in a case where determination is the fluorescence imagephotographing mode, the processing proceeds to step S304.

-   (Steps S302 to S303)

In a case where the control unit determined in step S301 that it is thevisible light image photographing mode, the imaging unit performed insteps S302 to S303 light emission processing necessary for visible lightimage photographing and visible light image photographing processingunder the control of the control unit.

-   (Steps S304 to S305)

In contrast, in a case where the control unit determined in step S301that it is the fluorescence image photographing mode, the imaging unitperforms in steps S304 to S305 light emission processing necessary forfluorescence image photographing and fluorescence image photographingprocessing under the control of the control unit.

-   (Step S306)

Next, in step S306, the control unit determines whether or not aconsecutive photographed image pair of a visible light image and afluorescence image has been obtained.

The timing of determination that the consecutive photographed image pairof the visible light image and the fluorescence image has been obtainedis a timing of acquisition of the fluorescence image (f4) illustrated inFIG. 15, for example.

In a case where it is not a timing of obtaining a consecutivephotographed image pair of the visible light image and the fluorescenceimage, the processing returns to step S301, and the processing of stepS301 and the subsequent steps is repeated.

In contrast, in a case where it is determined that the consecutivephotographed image pair of the visible light image and the fluorescenceimage has been obtained, the processing proceeds to step S307.

-   (Step S307)

Processing in steps S307 to S309 is image processing to be executed inthe image processing unit illustrated in FIGS. 2, 3, 5, or the like, forexample.

In step S307, feature amount extraction from the image is performed.

This processing is processing to be executed by the feature amountclassification processing unit illustrated in FIGS. 3, 5, or the like,for example.

The feature amount classification processing unit inputs the visiblelight image and the fluorescence image, extracts an image feature amountfrom these images, executes classification processing based on theextracted feature amount and stores data into a storage unit (database),while inputting a feature amount data classification result to the imagecorrection parameter calculation unit.

Note that the feature amount extracted from the image by the featureamount classification processing unit is, for example, a feature amountsuch as a point spread function (PSF) (=a function indicating a blurmode) being blur mode information.

Note that other feature amounts include, for example, luminancedistribution information and noise information, or the like as describedabove with reference to FIG. 4.

-   (Step S308)

Next, step S308 executes correction parameter calculation processing.

This processing is processing executed by the image correction parametercalculation unit illustrated in FIGS. 3, 5, or the like, for example.

The image correction parameter calculation unit uses the correction modedetermination information input from the feature amount classificationprocessing unit and training data stored in the storage unit (database)to determine the image correction parameter to be used for performingimage quality enhancement processing on the visible light image and thefluorescence image.

The determined image correction parameter is input to the imagecorrection unit.

Note that one specific example of the correction parameter calculated bythe image correction parameter calculation unit is a multiplicationcoefficient being a setting parameter of an inverse filter forsuppressing blur.

The image correction parameter calculation unit generates an inversefilter such as the Wiener filter to be applied for suppressing blur.

-   (Step S309)

Finally, step S309 executes image correction processing. This processingis processing executed by the image correction unit illustrated in FIGS.3, 5, or the like, for example.

The image correction unit applies the image correction parameter inputfrom the image correction parameter calculation unit to execute imagecorrection processing on the visible light image and the fluorescenceimage, and then, generates and outputs the corrected visible light imageand the corrected fluorescence image that have undergone image qualityenhancement processing.

Note that a specific example of the correction processing is pixel valuecorrection processing that utilizes an inverse filter such as a Wienerfilter that executes blur suppression processing.

For example, the blur suppression processing described above withreference to FIGS. 6 to 10 is executed.

The corrected image having blur suppressed and image quality enhanced bythis image correction processing is output.

Note that the processing example described with reference to FIGS. 6 to10 focused on the blur suppression processing on the fluorescence image,while it is also possible to suppress the blur of the visible lightimage with application of similar processing stages to the visible lightimage.

-   (Step S310)

Next, step S310 determines whether or not switching of the imagephotographing mode has occurred.

In a case where it is determined that switching of the imagephotographing mode has not occurred, that is, same type of imagephotographing is continuously performed, the processing returns to stepS309 and image correction is performed on the image. This imagecorrection is executed using the correction mode and the correctionparameter determined on the basis of the consecutive photographed imagepair of the visible light image and the fluorescence image obtained instep S306.

This processing corresponds to image correction processing for thefluorescence image (f5) to the fluorescence image (f7) illustrated inFIG. 15, for example.

Image correction on the fluorescence image (f5) to the fluorescenceimage (f7) illustrated in FIG. 15 is executed using the correction modeand the correction parameter determined on the basis of the visiblelight image (f3) and the fluorescence image (f4).

In a case where it is determined in step S310 that switching of theimage photographing mode has occurred, the processing returns to stepS301, and processing of step S301 and the subsequent steps are executed.

With the processing according to this flow, image correction can beperformed on all images even in a case where the image photographingsequence as illustrated in FIG. 15 described above has been executed.

-   [7. Hardware Configuration Example of Image Processing Apparatus]

Next, an example of the hardware configuration of the image processingapparatus will be described with reference to FIG. 19.

FIG. 19 is a diagram illustrating a hardware configuration example of animage processing apparatus that executes processing of the presentdisclosure.

A central processing unit (CPU) 601 functions as a control unit or adata processing unit that executes various types of processing inaccordance with a program stored in a read only memory (ROM) 602 or astorage unit 608. For example, the processing according to the sequencedescribed in the above exemplary embodiment is executed. A random accessmemory (RAM) 603 stores programs executed by the CPU 601, data, or thelike. The CPU 601, the ROM 602, and the RAM 603 are mutually connectedby a bus 604.

The CPU 601 is connected to an input/output interface 605 via the bus604. The input/output interface 605 is connected to an input unit 606that inputs a photographed image of an imaging unit 621, and includingvarious switches, a keyboard, a mouse, a microphone, or the like thatcan be used for user input, and also connected to an output unit 607that executes data output to a display unit 622, a speaker, or the like.The CPU 601 executes various types of processing in accordance with aninstruction input from the input unit 606, and outputs processingresults to the output unit 607, for example.

The storage unit 608 connected to the input/output interface 605includes a hard disk or the like, for example, and stores a program tobe executed by the CPU 601 and various data. A communication unit 609functions as a transmission/reception unit for Wi-Fi communication,Bluetooth (registered trademark) communication, and other datacommunication via a network such as the Internet, a local area network,or the like, and communicates with an external apparatus.

A drive 610 connected to the input/output interface 605 drives aremovable medium 611 such as a magnetic disk, an optical disk, amagneto-optical disk or a semiconductor memory such as a memory card orthe like, and executes data recording or reading.

-   [8. Summary of Configuration Of Present Disclosure]

The exemplary embodiments of the present disclosure have been describedin detail with reference to specific exemplary embodiments as above.Still, it is self-evident that those skilled in the art can makemodifications and substitutions of the exemplary embodiments withoutdeparting from the scope and spirit of the present disclosure. In otherwords, the present invention has been disclosed in the form ofexemplification, and should not be interpreted restrictively. In orderto judge the scope of the present disclosure, the section of claimsshould be taken into consideration.

Note that the technology disclosed in the present description can beconfigured as follows.

-   (1) An image processing apparatus including:

a feature amount classification processing unit that inputs afluorescence image and a visible light image and extracts a featureamount from at least one of the images; and

an image correction unit that executes pixel value correction processingon the fluorescence image on the basis of a correction parameterdetermined in accordance with the feature amount.

-   (2) The image processing apparatus according to (1), further    including a correction parameter calculation unit that determines a    correction parameter to be used for pixel value correction in the    image correction unit on the basis of the feature amount,

in which the image correction unit

executes pixel value correction processing applying a correctionparameter determined by the correction parameter calculation unit.

-   (3) The image processing apparatus according to (1) or (2),

in which the feature amount classification processing unit

obtains blur mode information from the fluorescence image, and

the image correction unit

executes pixel value correction processing on the fluorescence image soas to reduce blur of the fluorescence image.

-   (4) The image processing apparatus according to (3), further    including a correction parameter calculation unit that determines a    correction parameter to be used for pixel value correction in the    image correction unit on the basis of the feature amount,

in which the correction parameter calculation unit determines acorrection parameter to suppress blur of the fluorescence image, and

the image correction unit

executes pixel value correction processing applying a correctionparameter determined by the correction parameter calculation unit.

-   (5) The image processing apparatus according to any of (1) to (4),    in which the feature amount classification processing unit extracts    one feature amount out of (a) to (c):

(a) luminance distribution information;

(b) blur mode information; and

(c) noise information,

from at least one of the fluorescence image or the visible light image.

-   (6) The image processing apparatus according to any of (1) to (5),    further including an image separating unit that inputs a mixed image    of a visible light image and a fluorescence image and generates a    visible light image and a fluorescence image from the mixed image,

in which the feature amount classification processing unit

inputs the fluorescence image and the visible light image generated bythe image separating unit, and

the image correction unit

executes pixel value correction processing based on a correctionparameter determined in accordance with the feature amount, onto thefluorescence image generated by the image separating unit.

-   (7) The image processing apparatus according to any of (1) to (6),

further including

an image separating unit that alternately inputs a visible light imageand a fluorescence image, separates an input image into a visible lightimage and a fluorescence image and outputs the images,

in which the feature amount classification processing unit

inputs the fluorescence image and the visible light image output by theimage separating unit, and

the image correction unit

executes pixel value correction processing based on a correctionparameter determined in accordance with the feature amount, onto thefluorescence image output by the image separating unit.

-   (8) The image processing apparatus according to any of (1) to (7),

in which, in a case where fluorescence images are consecutive as theinput image,

the image correction unit

-   applies a correction parameter determined on the basis of a feature    amount obtained by application of a consecutive photographed image    pair of a visible light image and a fluorescence image that are    closest in time in the past so as to execute correction processing    of the consecutively input fluorescence images.-   (9) The image processing apparatus according to any of (1) to (8),

further including an interpolation image generation unit that generatesa virtual fluorescence image at non-photographing timing of fluorescenceimages on the basis of preceding and succeeding photographed images,

in which the feature amount classification processing unit

inputs the interpolation image generated by the interpolation imagegeneration unit to execute feature amount extraction processing, and

the image correction unit

executes pixel value correction processing based on a correctionparameter determined in accordance with the feature amount, onto theinterpolation image generated by the interpolation image generationunit.

-   (10) The image processing apparatus according to any of (1) to (9),

in which the image correction unit

executes pixel value correction processing on the visible light image onthe basis of a correction parameter determined in accordance with thefeature amount.

-   (11) An imaging apparatus including:

an imaging unit that performs imaging processing of a visible lightimage and a fluorescence image, or a visible-fluorescence mixture image;

an image separating unit that inputs a photographed image of the imagingunit, separates a visible light image and a fluorescence image from theinput image and outputs the separated images;

a feature amount classification processing unit that inputs thefluorescence image and the visible light image output by the imageseparating unit and extracts a feature amount from at least one of theimages; and

an image correction unit that executes pixel value correction processingon the fluorescence image output by the image separating unit on thebasis of a correction parameter determined in accordance with thefeature amount.

-   (12) The imaging apparatus according to (11), further including a    correction parameter calculation unit that determines a correction    parameter to be used for pixel value correction in the image    correction unit on the basis of the feature amount,

in which the image correction unit

executes pixel value correction processing applying a correctionparameter determined by the correction parameter calculation unit.

-   (13) The imaging apparatus according to (11) or (12) claim 11,

in which the feature amount classification processing unit

obtains blur mode information from the fluorescence image, and

the image correction unit

executes pixel value correction processing on the fluorescence image soas to reduce blur of the fluorescence image.

-   (14) The imaging apparatus according to (13), further including a    correction parameter calculation unit that determines a correction    parameter to be used for pixel value correction in the image    correction unit on the basis of the feature amount,

in which the correction parameter calculation unit determines acorrection parameter to suppress blur of the fluorescence image, and

the image correction unit

executes pixel value correction processing applying a correctionparameter determined by the correction parameter calculation unit.

-   (15) An image processing method executed in an image processing    apparatus, the image processing method including executing:

a feature amount calculation step of executing, by a feature amountclassification processing unit, input of a fluorescence image and avisible light image and extraction of a feature amount from at least oneof the images; and

an image correction step of executing, by an image correction unit,pixel value correction processing on the fluorescence image on the basisof a correction parameter determined in accordance with the featureamount.

-   (16) A program that causes an image processing apparatus to execute    image processing, the image processing including processing of:

causing a feature amount classification processing unit to input afluorescence image and a visible light image and extract a featureamount from at least one of the images; and

causing an image correction unit to execute pixel value correctionprocessing on the fluorescence image on the basis of a correctionparameter determined in accordance with the feature amount.

Furthermore, the series of processing described in the description canbe executed by hardware, software, or a combination of both. In the caseof executing the processing by software, it is possible to allow theprogram recording processing sequences to be installed and executed on amemory within a computer, incorporated in dedicated hardware, orpossible to allow the program to be installed and executed on ageneral-purpose computer capable of executing various types ofprocessing. For example, the program can be recorded in a recordingmedium beforehand. The program can be installed from a recording mediumto a computer, or can be received via a network such as a local areanetwork (LAN) or the Internet so as to be installed in a recordingmedium such as a built-in hard disk.

Note that the various types of processing described in the descriptionmay be executed in parallel or separately in accordance with theprocessing capability of the apparatus that executes the processing orin accordance with necessity, in addition to execution in time seriesfollowing the description. Note that in the present description, thesystem represents a logical set of a plurality of apparatuses, and thatall the constituent apparatuses need not be in a same housing.

INDUSTRIAL APPLICABILITY

As described above, according to a configuration of one exemplaryembodiment of the present disclosure, it is possible to implement anapparatus and a method to execute image quality enhancement processingon fluorescence images.

Specifically, the fluorescence image and the visible light image areinput and the image feature amount is extracted, and pixel valuecorrection processing is executed on the fluorescence image on the basisof a correction parameter determined in accordance with the featureamount. The correction parameter used for pixel value correction isdetermined by the correction parameter calculation unit on the basis ofthe feature amount. The image correction unit executes pixel valuecorrection processing that applies the correction parameter determinedby the correction parameter calculation unit. For example, blur modeinformation is obtained as a feature amount from a fluorescence image,and the image correction unit executes pixel value correction processingon the fluorescence image so as to reduce blur of the fluorescenceimage.

This processing enables implementation of an apparatus and a method forexecuting image quality enhancement processing on a fluorescence image.

REFERENCE SIGNS LIST

-   10 Living tissue-   11Blood vessel-   100 Image processing apparatus-   101 Control unit-   102 Storage unit-   103 Codec-   104 Input unit-   105 Output unit-   106 Imaging unit-   151 Corrected visible light image-   152 Corrected fluorescence image-   301 Image separating unit-   302 Feature amount classification processing unit-   303 Storage unit (database)-   304 Image correction parameter calculation unit-   305 Image correction unit-   321 Image separating unit-   322 Feature amount classification processing unit-   333 Storage unit (database)-   324 Image correction parameter calculation unit-   325 Image correction unit-   330 PSF estimation unit-   340 Inverse filter calculation unit-   350 PSF estimation unit-   360 Inverse filter calculation unit-   400 Image processing apparatus-   410 Interpolation image generation unit-   411 Motion estimation processing (ME) unit-   412 Interpolation image estimation unit-   413 Subtraction unit-   450 Image processing unit-   451 Feature amount classification processing unit-   452 Storage unit (database)-   453 Image correction parameter calculation unit-   454 Image correction unit-   501 Image feature amount classification processing unit-   502 Image correction parameter calculation unit-   503 Image correction unit-   601 CPU-   602 ROM-   603 RAM-   604 Bus-   605 Input/output interface-   606 Input unit-   607 Output unit-   608 Storage unit-   609 Communication unit-   610 Drive-   611 Removable medium-   621 Imaging unit-   622 Display unit

1. An image processing apparatus comprising: a feature amountclassification processing unit that inputs a fluorescence image and avisible light image and extracts a feature amount from at least one ofthe images; and an image correction unit that executes pixel valuecorrection processing on the fluorescence image on the basis of acorrection parameter determined in accordance with the feature amount.2. The image processing apparatus according to claim 1, furthercomprising a correction parameter calculation unit that determines acorrection parameter to be used for pixel value correction in the imagecorrection unit on the basis of the feature amount, wherein the imagecorrection unit executes pixel value correction processing applying acorrection parameter determined by the correction parameter calculationunit.
 3. The image processing apparatus according to claim 1, whereinthe feature amount classification processing unit obtains blur modeinformation from the fluorescence image, and the image correction unitexecutes pixel value correction processing on the fluorescence image soas to reduce blur of the fluorescence image.
 4. The image processingapparatus according to claim 3, further comprising a correctionparameter calculation unit that determines a correction parameter to beused for pixel value correction in the image correction unit on thebasis of the feature amount, wherein the correction parametercalculation unit determines a correction parameter to suppress blur ofthe fluorescence image, and the image correction unit executes pixelvalue correction processing applying a correction parameter determinedby the correction parameter calculation unit.
 5. The image processingapparatus according to claim 1, wherein the feature amountclassification processing unit extracts one feature amount out of (a) to(c): (a) luminance distribution information; (b) blur mode information;and (c) noise information, from at least one of the fluorescence imageor the visible light image.
 6. The image processing apparatus accordingto claim 1, further comprising an image separating unit that inputs amixed image of a visible light image and a fluorescence image andgenerates a visible light image and a fluorescence image from the mixedimage, wherein the feature amount classification processing unit inputsthe fluorescence image and the visible light image generated by theimage separating unit, and the image correction unit executes pixelvalue correction processing based on a correction parameter determinedin accordance with the feature amount, onto the fluorescence imagegenerated by the image separating unit.
 7. The image processingapparatus according to claim 1, further comprising an image separatingunit that alternately inputs a visible light image and a fluorescenceimage, separates an input image into a visible light image and afluorescence image and outputs the images, wherein the feature amountclassification processing unit inputs the fluorescence image and thevisible light image output by the image separating unit, and the imagecorrection unit executes pixel value correction processing based on acorrection parameter determined in accordance with the feature amount,onto the fluorescence image output by the image separating unit.
 8. Theimage processing apparatus according to claim 1, wherein, in a casewhere fluorescence images are consecutive as the input image, the imagecorrection unit applies a correction parameter determined on the basisof a feature amount obtained by application of a consecutivephotographed image pair of a visible light image and a fluorescenceimage that are closest in time in the past so as to execute correctionprocessing of the consecutively input fluorescence images.
 9. The imageprocessing apparatus according to claim 1, further comprising aninterpolation image generation unit that generates a virtualfluorescence image at non-photographing timing of fluorescence images onthe basis of preceding and succeeding photographed images, wherein thefeature amount classification processing unit inputs the interpolationimage generated by the interpolation image generation unit to executefeature amount extraction processing, and the image correction unitexecutes pixel value correction processing based on a correctionparameter determined in accordance with the feature amount, onto theinterpolation image generated by the interpolation image generationunit.
 10. The image processing apparatus according to claim 1, whereinthe image correction unit executes pixel value correction processing onthe visible light image on the basis of a correction parameterdetermined in accordance with the feature amount.
 11. An imagingapparatus comprising: an imaging unit that performs imaging processingof a visible light image and a fluorescence image, or avisible-fluorescence mixture image; an image separating unit that inputsa photographed image of the imaging unit, separates a visible lightimage and a fluorescence image from the input image and outputs theseparated images; a feature amount classification processing unit thatinputs the fluorescence image and the visible light image output by theimage separating unit and extracts a feature amount from at least one ofthe images; and an image correction unit that executes pixel valuecorrection processing on the fluorescence image output by the imageseparating unit on the basis of a correction parameter determined inaccordance with the feature amount.
 12. The imaging apparatus accordingto claim 11, further comprising a correction parameter calculation unitthat determines a correction parameter to be used for pixel valuecorrection in the image correction unit on the basis of the featureamount, wherein the image correction unit executes pixel valuecorrection processing applying a correction parameter determined by thecorrection parameter calculation unit.
 13. The imaging apparatusaccording to claim 11, wherein the feature amount classificationprocessing unit obtains blur mode information from the fluorescenceimage, and the image correction unit executes pixel value correctionprocessing on the fluorescence image so as to reduce blur of thefluorescence image.
 14. The imaging apparatus according to claim 13,further comprising a correction parameter calculation unit thatdetermines a correction parameter to be used for pixel value correctionin the image correction unit on the basis of the feature amount, whereinthe correction parameter calculation unit determines a correctionparameter to suppress blur of the fluorescence image, and the imagecorrection unit executes pixel value correction processing applying acorrection parameter determined by the correction parameter calculationunit.
 15. An image processing method executed in an image processingapparatus, the image processing method comprising executing: a featureamount calculation step of executing, by a feature amount classificationprocessing unit, input of a fluorescence image and a visible light imageand extraction of a feature amount from at least one of the images; andan image correction step of executing, by an image correction unit,pixel value correction processing on the fluorescence image on the basisof a correction parameter determined in accordance with the featureamount.
 16. A program that causes an image processing apparatus toexecute image processing, the image processing comprising processing of:causing a feature amount classification processing unit to input afluorescence image and a visible light image and extract a featureamount from at least one of the images; and causing an image correctionunit to execute pixel value correction processing on the fluorescenceimage on the basis of a correction parameter determined in accordancewith the feature amount.